Deprivation: “a state of exclusion from the ordinary customs and activities of society.”
The Annual Deprivation Index (ADI) is an up to date data source on core deprivation metrics in the cities, towns and regions of England.
Recently published on 3 March 2023 by the organisation Autonomy in conjunction with academics. Working paper for data release can be found here.
The ADI is constructed using granular, high-frequency indicators in 3 domains (see tables below):
mental health (sub-domains of NHS quality outcomes framework)
employment (job claimant count data)
crime (various sub-domains, inc. burglary, vehicle crime, violent crime, etc.)
The ADI is a summation of these factors which, when combined with data on population per area, allows us to calculate a rate (ADI cases/person) for each LSOA or District (many LSOAS to one District).
It’s a cardinal scale, so the level of measurement is numeric and not only a ranking (ordinal), like the Index of Multiple Deprivation (IMD)
| area_code | area_name | pop | cases_claims | rate_claims | cases_crime | rate_crime | cases_health | rate_health | cases_ADI_LSOA | rate_ADI_LSOA | year |
|---|---|---|---|---|---|---|---|---|---|---|---|
| E01000001 | City of London 001A | 1,102 | 70 | 0.06 | 234 | 0.21 | 51 | 0.05 | 355 | 0.32 | 2013 |
| E01000002 | City of London 001B | 1,074 | 110 | 0.10 | 353 | 0.33 | 53 | 0.05 | 516 | 0.48 | 2013 |
| E01000003 | City of London 001C | 985 | 345 | 0.35 | 69 | 0.07 | 45 | 0.05 | 459 | 0.47 | 2013 |
| E01000005 | City of London 001E | 693 | 465 | 0.67 | 596 | 0.86 | 37 | 0.05 | 1,098 | 1.59 | 2013 |
| E01000006 | Barking and Dagenham 016A | 1,410 | 555 | 0.39 | 105 | 0.07 | 102 | 0.07 | 762 | 0.54 | 2013 |
| E01000007 | Barking and Dagenham 015A | 1,114 | 1,230 | 1.10 | 627 | 0.56 | 82 | 0.07 | 1,939 | 1.74 | 2013 |
| district | pop | cases_ADI | rate_ADI | cases_claims | rate_claims | cases_crime | rate_crime | cases_health | rate_health | year |
|---|---|---|---|---|---|---|---|---|---|---|
| City of London | 5,342 | 9,435 | 1.77 | 1,260 | 0.24 | 7,916 | 1.48 | 259 | 0.05 | 2013 |
| Barking and Dagenham | 142,423 | 110,809 | 0.78 | 77,310 | 0.54 | 23,406 | 0.16 | 10,093 | 0.07 | 2013 |
| Barnet | 290,457 | 122,196 | 0.42 | 72,830 | 0.25 | 33,483 | 0.12 | 15,883 | 0.05 | 2013 |
| Bexley | 188,214 | 77,105 | 0.41 | 49,005 | 0.26 | 17,928 | 0.10 | 10,172 | 0.05 | 2013 |
| Bromley | 255,478 | 103,496 | 0.41 | 57,535 | 0.23 | 27,554 | 0.11 | 18,407 | 0.07 | 2013 |
| Brent | 252,001 | 161,200 | 0.64 | 109,725 | 0.44 | 34,383 | 0.14 | 17,092 | 0.07 | 2013 |
ADI vs IMD:
Like IMD, it contains granular data at Lower Super Output Area (LSOA) levels
Monitor indicators more frequently to gain more real-time insights; i.e., annually vs IMD which is calculated approx. every 5 years
Because ADI is cardinal, we can measure change in absolute levels of deprivation over time
We can also measure levels of and changes in inequality
Here, I’m exploring inequality in the ADI and seeing whether it’s useful for insights at “hyper-local” level (and e.g., useful in MRP).
There are almost 33,000 LSOAs, grouped into 326 Districts.
caveat: ADI uses statistical (geographic, produced by the ONS) and not administrative (political ward) boundaries, and I haven’t yet mapped LSOAs to wards, which change over time – so no voting insights yet.
All source data from ADI website, but ADI construction and analysis is my own (and so my own errors and imputations when data was missing!).
The figure below shows ADI rate average and distribution over time. There are two broad features that stand out which I’ll dive into:
Significant decrease and then increase over time
Initial squashing and then dispersion of distribution
Take a look at number of ADI cases over time, overall and by sub-category.
First, where the various sub-categories were initially converging, the bulk of ADI cases in the last few years are made up of job claims (stacked graph shows proportion contributions).
Second, we can see the enormous impact of COVID, primarily on job claims rather than the other sub-categories. Job claims data shows the largest variation as well.
Third, though starting from a low base, there’s been an almost doubling of mental health cases in the past decade.
Fourth, cases of crime have remained mostly the same over time.
The variation in and differences between sub-category contributions suggests that granular data on each could be a useful inclusion in predictive analysis, e.g., MRP.
Having looked at overall numbers, we can explore inequality in ADI rates. One approach is to separate LSOAs into groups with different deprivation levels.
I’ve ranked all LSOAs by ADI rate (not ADI cases!) from highest rate to lowest rate and then split them into ten equal-sized population groups (keeping in mind that LSOA populations vary). This gives us deciles ranging from 1: most deprived to 10: least deprived (as is the custom order).
The figure below plots the average ADI rate for each decile. Notice several things:
Wide gaps between the worst-off deciles and the rest.
Only small differences between the 5 best-off deciles, but gaps have recently appeared between them.
One impact of COVID is the recent dispersion of ADI rates—splaying out of sorts.
Something else we can see is that while the direction of rate changes was initially similar across deciles, they are now are at odds: the most deprived getting worse off while least deprived showing improvements.
The two graphs below show more clearly
the magnitude differences of ADI rate changes and
how important it is to take the the starting level into account.
Established that there
is significant variation in the ADI over time,
are important differences in changes by sub-category, and
recently, ADI rates have risen for the most deprived while they’ve decreased for the most well-off
One direction to explore further is to more directly measure inequality in the ADI.
Some background of how inequality is usually measured might be useful.
There are different ways we can measure inequality. Most often researchers use data on people’s income (we could use wages, total assets, etc.) and compare the different ends of the distribution.
The figure below shows median income levels by income quintile (i.e., after ranking all individuals from lowest to highest income and partitioning them into five equal-sized groups).
We can eyeball the differences between groups and get broad insights: e.g., median income of the top quintile is over 4x higher than the bottom.
A popular alternative is to draw a Lorenz curve and calculate a Gini coefficient from the underlying data.
The Lorenz curve plots the cumulative proportion of total income by the cumulative proportion of the population, with the latter ranked from lowest to highest (often grouped, here into percentiles).
The figure usually includes a diagonal line of perfect equality, representing the case where each population group earns their equivalent proportion of total income.
Reading along the curve tells us the proportion of income that is earned by the poorest X% of the population. Hovering over the graph below (which is based on 2019 UK income data), the poorest half (50%) of the population earn around 27% of all income.
The Gini coefficient is a measure of inequality based on this data. It roughly measures the percentage difference between the area under the line of perfect equality and the area under the Lorenz curve.
The intuition is that the further ‘away’ the Lorenz curve is from the line of perfect equality—so, the smaller the area under the curve—the more unequal the distribution is and the greater the difference will be … i.e. the ‘flatter-then-steeper’ the curve, the lower the proportion of total income received by the poorest groups.
A higher Gini coefficient thus indicates greater income inequality.
Though widely used, there are several cons with using the curve, e.g., plotting multiple Lorenz curves on the same figure can make changes or differences between groups hard to interpret.
Still, it’s useful to measure changes in inequality over time, as in the figure below. We can see income inequality rising from the 80s until around the time of the financial crisis, and then slowly falling.
The figure also broadly hints at the role of public transfers in reducing inequality, with a slight widening in the difference between disposable and gross (after transfers) income.
We must be careful though, as the Gini coefficient alone may not be very sensitive. The figure below shows the Gini coefficient on disposable income from the past three years, with 95% confidence intervals representing uncertainty in estimates of mean and median income. There’s a significant overlap, so conclusions about changes must be tempered!
As we did for income, we can calculate and plot a Lorenz curve for ADI cases across LSOAs. There’s an important difference, though.
As shown in the decile calculations earlier, deprivation measures typically rank from most to least deprived, rather than least to most as with income. So less well-off areas will have higher ADI rates and more cases than those better-off, and each additional area adds fewer cases than the previous—meaning the slope of the curve is diminishing rather than increasing.
But the figure can still be read in the same way. We see, for example, that the 20% worst-off areas make up almost 39% of all ADI cases, and that the most deprived half (50%) make up almost 72% of ADI cases.
We can take the 2021 data and calculate the Gini coefficient for each sub-category separately. Inequality in job claims shows the greatest deviation from perfect equality, followed by crime cases. There is remarkably low inequality in mental health cases across all LSOAs!
And we can calculate the Gini coefficient over time as well, for overall ADI and each sub-category. We see that inequality in mental health has always been low (in our data) and inequality in job claims the highest—though the latter has recently converged with inequality in crime cases.
Plotting ADI alongside income inequality in the figure below, we can see that they track each other reasonably well but diverge more recently.
This suggests that the ADI measure has some construct validity (truly measures what we care about)—see the working paper for its relation to IMD measures.
It also suggests that the ADI offers some additional variation that could provide new insight that we didn’t have before from measuring e.g. income alone.
We’ve looked broadly at ADI cases / rate changes over time and inequality among LSOAs, both overall and by sub-category. It’s possible to get more granular and look more closely at ADI changes in specific areas.
For example, the table below shows data at District level (LSOAs are grouped into their respective Districts).
The orange column shows each district’s change in ADI rate between 2020 and 2021. We can sort the table by this column to see Districts with the largest positive changes (indicates a worsening) or negative changes (indicates an improvement) in ADI rate.
There’s a lot to explore. To start, we don’t yet know what a particularly good or bad rate change is.
We can plot rate changes by districts (ranked from worst to best), as in the figure below, and see if there are any patterns (red line: reverse of the typical inverse of the lognormal cumulative distribution function [??]).
As in the figures below, we can arbitrarily select and analyse the top 10 worst and best performing districts and see how they’ve changed relative to others.
One interesting insight is that, there’s a large overlap between the best and worst group’s 2020 starting points; i.e., they are reasonably spread across the distribution.
Why do districts at similar levels see such vastly different changes? A question for the future!
Finally, given the variation of Districts’ experiences, another interesting question is to what extent the inequality in ADI rates is driven by differences between districts or within the districts themselves.
This approach has more general application: is inequality greater between or within rural vs urban spaces, or within or between race and gender groupings?
Thankfully, there’s a measure of inequality that’s suitable for this: Theil’s generalised entropy measure of inequality can be broken into a between- and within-group component of inequality.
The formula looks like this:
Plotting the Theil total measure of ADI inequality over time looks similar to the Gini plot earlier (good news suggesting the calculations are correct!).
We also see that overall inequality in ADI looks driven by within-district rather than between-district inequality.
This probably has important implications for how the sub-categories of ADI can be addressed—and for political analysis that focuses on micro-level experiences between geographically close areas.